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ABSTRACT 

We explored the clustering properties of Lyman break galaxies (LBGs) at z — 4 and 5 with an 
angular two-point correlation function on the basis of the very deep and wide Subaru Deep Field 
data. We confirmed the previous result that the clustering strength of LBGs depends on the UV 
luminosity in the sense that brighter LBGs are more strongly clustered. In addition, we found an 
apparent dependence of the correlation function slope on UV luminosity for LBGs at both z = 4 and 
5. More luminous LBGs have a steeper correlation function. The bias parameter was found to be 
a scale-dependent function for bright LBGs, whereas it appears to be almost scale-independent for 
faint LBGs. Luminous LBGs have a higher bias at smaller angular scales, which decreases as the scale 
increases. To compare these observational results, we constructed numerical mock LBG catalogs based 
on a semianalytic model of hierarchical clustering combined with high-resolution Y-body simulation, 
carefully mimicking the observational selection effects. The luminosity functions and the overall 
correlation functions for LBGs at z = 4 and 5 predicted by this mock catalog were found to be almost 
consistent with the observation. The observed dependence of the clustering on UV luminosity was not 
reproduced by the model, unless subsamples of distinct halo mass were considered. That is, LBGs 
belonging to more massive dark halos had steeper and larger amplitude correlation functions. With 
this model, we found that LBG multiplicity in massive dark halos amplifies the clustering strength 
at small scales, which steepens the correlation function. The hierarchical clustering model could 
therefore be reconciled with the observed luminosity dependence of the correlation function if there is 
a tight correlation between UV luminosity and halo mass. Our finding that the slope of the correlation 
function depends on luminosity could be an indication that massive dark halos hosted multiple bright 
LBGs. 

Subject headings: galaxies: high-redshift — cosmology: observations — cosmology: theory — large- 
scale structure of universe 
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1. INTRODUCTION 

The clustering strength of galaxies is one of the most 
fundamental measures in observational cosmology. The 
large-scale structure observed today was formed by the 
gravitational growth of initial small density fluctuations. 
The measurement of galaxy clustering and its evolu- 
tion enables us to see the history of galaxy assembly, 
which is an essential process in the present-day standard 
paradigm of hierarchical galaxy formation. However, it 
is not so straightforward that clustering measurements 
alone can provide a direct determination of essential cos- 
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mological parameters or the initial power spectrum. This 
is because clustering measurements can be made only 
with observable galaxies, not with the invisible dark mat- 
ter halos, whose mass dominates the universe and drives 
gravitational interaction. The formation process of ob- 
servable galaxies in these dark matter halos remains un- 
certain, and is an important outstanding problem in the 
interpretation of observed galaxy clustering. The forma- 
tion of galaxies depends strongly on complicated physical 
processes connected to the evolution of the baryonic com- 
ponents, such as gas cooling, star formation, supernova 
feedback in each dark halo, and galaxy mergers. We have 
to infer how galaxies trace the underlying dark matter 
throughout their evolution with cosmic time, for which 
purpose both detailed observations and comparison with 
theoretical modeling are essential. 

The gravitational clustering evolution of dark mat- 
ter from specified initial conditions has been thoroughly 
investigated in the hierarchical formation picture using 
both iV-body simulations and analytic methods. Semi- 
analytic approaches have been successful in modeling 
complex galaxy formation processes. Advances in com- 
puter technology have enabled precise cosmological pre- 
dictions to be made based on the use of more robust 
frameworks with larger volumes and finer resolution. 
Indeed, in the local universe, detailed comparisons of 
galaxy clustering for various subsets of galaxies have 
been made for these models and the precise observa- 
tions provided by the Sloan Digital Sky Survey (SDSS; 
iZehavi et ail 120021 and th e Two Degree Field Galaxy 
Redshift Survey (2dFGRS; IKTorberg et al]l20f)l . In ad- 
dition to the previously known dependence of correlation 
amplitude on luminosity, color, morphology, and spectral 
type, more detailed signatures such as the slight slope 
differences among different subsamples and a systematic 
departure from a single power law have been revealed 
(|Zehavi et al J EM 1200^1 . A similarly precise measure- 
ment of clustering is required for the high-z universe to 
make a detailed comparison with the model predictions, 
which is now feasible. 

On the theoretical side, clustering measurements 
in the high-z universe were initially thought to be 
a valuable tool in distinguishing cosmological mod- 
els. The observed strong clustering with high bias 
at z > 3, however, can be easily reproduced in most 
of the cold dark matter (CDM ) cosmologie s with 
either semiana l ytic ap proaches ([BlaizojLel_aL| I2HQJ; 
iWech sler et al.l 120011 iSommer ville. P rimack fc Faberl 
120011 iGovernato et alJ 119981 iBaugh et all 119981) or hv- 
drodynamic simulations tffieinberg^Her nauist fc Katd 
120021 iKatz. Hernauist fc Weinberg! H999jT Since the 
precise measurement by the Wilkinson Microwave 
Anisotropy Probe ( WMAP) of the cosmic microwave 
background (CMB) has established an appropriate cos- 
mological model, clustering measurements have instead 
turned out to be an effectiv e tool for constraining the na- 
ture of LBGs themselves l)Katz. Hernauist fc Weinberel 
1999). As in the case of the local universe, a detailed 
study of clustering properties and their dependence on 
high-z galaxy properties provides hints for understand- 
ing what determines these properties and how galaxies 
trace the underlying dark matter distribution. 

Multicolor selection technique has efficiently revealed 
an abundant population of high-z star-forming galaxies, 



called Lyman break galaxies (LBGs: ISteidel et ali l9961. 
at z > 2. It has been shown that this high-z popula- 
tion has a large clustering length (rn = 3 — 6/i -1 Mpc; 
iGiavalisco et al.lll998l iFoucaud et alJl2003|) comparable 
to that of present-day bright galaxies. This result sug- 
gested a comparatively large value of the bias factor at 
z ~ 3, which has generally been interpreted as indirect 
evidence that LBGs resi de in massive dark mat t er ha- 
los with M > 10 n M o . IGiavalisco fc Dickinson! <|2001fl 
found an additional important tendency that more lumi- 
nous LBGs have higher clustering strengths. This implies 
that the luminosity (in this case, UV luminosity, which 
corresponds to star formation rate) of LBGs is governed 
by the halo mass of the galaxy. The same attempts to 
quantify the amo unt of clustering hav e been applied to 
LBGs at higher z (Ouc hi et al.ll2004m . and the strength 
was found to be almost constant over the range z = 3 — 
5, which means high-bias galaxy formation during early 
epochs. The observed clustering strength constrains the 
dark halo mass of LBGs by comparison with ei ther ana- 
lytic treatments (Moustakas & Somerville 2002) or semi- 
analytic models l)Blaizot et al J I20041 . 

However, the fields of view (FOVs) of these surveys 
to probe high-z galaxies are generally small, excep t in a 
few cases ilOuchi et alJl2004atlFoucaud et alJl2003^ . The 
small FOV leads to a systematic underestimation of the 
clustering scale and limits a statistically robust analy- 
sis against cosmic variance, which crucially affects the 
estimate of the normalization of the correlation func- 
tion. Small-scale measurements also fail to provide a 
true estimate of the shape of the correlation function, 
which is generally assumed to be a single power law 
£(r) = (r/r )~ J with a slope parameter 7 = 1.8. The 
slope parameter is much more loosely constrained for 
high-z LBGs, 7 = 2 ± .7 l|Giavalisco et alJ 11993 
IGiavalisco fc Dickinson! 120011) . compared with the lo- 
cal value of 7 = 1.71 ± 0.06 iNorberg et alJl200l|) or 
7 = 1.75 ± 0.03 (jZehavi et al.ll2002j) . In addition, there 
is an important implication that some dark halos may 
harbor more tha n one LBG, as suggested by numer- 
ical simulations ( Sommer ville. Primack fc Faberl 120011 
IWechsler et alJl200H) . If this is the case, then clustering 
analyses made using a small FOV are biased to probe 
dominantly for galaxy pairs in the same halo and will 
tend to miss the halo-halo pair contribution, which has 
a distinct clustering strength at large scales. 

We carried out a clustering analysis exploiting the very 
deep and wide imaging data of the Subaru Deep Field 
(SDF), which has an effective area of 876 arcmin 2 . Our 
previous study on LBG cluste ring in the SDF was pre- 
sented in Ouchi et al. (20042). The present paper re- 
ports an extensive analysis based on deeper images. The 
deepness of the SDF allows us to explore the faint LBG 
population with M\jv = — 19 + 5 1og/t7n at z = 4 an d 
M uv = -20 + 51og/i 70 at z = 5 l|Yoshida et a l.1 12006ft . 
which are comparable to the faintest LBGs at z = 3 
l)Oiavalisco fc DickinsonlfeOOlft . The SDF surveyed area 
is 5 times larger than a single Gr eat Observatories Ori- 
gins Deep Survey (GOODS) field (IGiavalisco e t al. 2004) 
and 250 times larger than the Hubble Deep Field (HDF). 
The clu stering analysis of th e Canada-France deep fields 
survey ijFoucaud et al.ll2003f ) was based on a fairly large 
contiguous field (28' x 28'); however, their limiting mag- 
nitude with a 4 m telescope is ~ 2 mag shallower than 
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ours. The details of the luminosity dependence of the 
galaxy clustering have been difficult to establish owing 
to the narrow luminosity range probed thus far in the 
high-z universe. In this study, we make a detailed analy- 
sis of the luminosity dependence of LBG clustering based 
on deep and wide SDF data. 

Furthermore, we directly compare the observed angu- 
lar correlation functions at z = 4 and 5 with theoreti- 
cal predictions given by a semianalytic model + iV-body 
simulation based on the hierarchical formation scheme. 
Mock LBG catalogs were compiled from the simulation 
data with the same selection criteria as in the real obser- 
vational sample. This is the first study to make such a 
comparison of LBG clustering properties at z = 4 and 5 
between observations and the simulations, and our deep 
and wide data allow a precise comparison. 

The outline of the paper is as follows. In § 2, we de- 
scribe our LBG sample. In § 3, we briefly review the 
model that we used and show that it predicts the same 
abundances of LBGs as the observations at z = 4 and 
5. The sky distribution of our observed LBG sample is 
shown in § 4. We present our results on the clustering 
analysis of the sample in § 5. A comparison between the 
observational results and the model predictions is made 
in § 6, where we also discuss the possible interpretation 
of our results. Our results are discussed further in § 7, 
and conclusions are presented in § 8. 

Throughout the paper, we analyze in the flat ACDM 
model: fi m = 0.3, ft a = 0.7, a 8 = 0.9, and the spec- 
trum parameter T = Q m h — 0.21, where the Hubble 
constant is defined as Ho = 70/i7okms~ 1 Mpc~ 1 . These 
parameters are consistent with recent CMB constraints 
j|Spergel et ail 12003ft . Magnitudes are given in the AB 
system. 

2. THE DATA AND THE LBG SAMPLE 

We used a set of very wide and deep multi-color imag- 
ing catalogs of the SDF 20 . The observations, the data 
processing, and the source detections are described in 
iKashikawa et all l)2004lh The 3a limiting magnitudes in 
2" apertures are 28.45, 27.74, 27.80, 27.43, and 26.62 in 
B, V, R, i', and z', respectively. The final co-added im- 
age has an effective area of 876 arcmin 2 with a unified 
seeing size of 0".9 for each band. The wide FOV enables 
us to probe large comoving volumes of 8.844 x 10 6 and 
5.390 x 10 6 /if 3 Mpc 3 for our LBG samples at (z) ~ 4 and 
~ 5, respectively. The absolute error of the astrometry 
was found to be as small as 0". 21-0". 27. 

We used the two high-z LBG samples presented in 
lYoshida et alJ l)2006lk in which full details of sample se- 
lection, completeness, and contamination rates are de- 
scribed. Here we briefly describe our LBG samples. 
These samples were constructed through a color selec- 
tion technique that distinguishes high-z galaxies having 
the strong Lyman break on their spectral energy distri- 
butions (SEDs). We used the (B - R) versus {R - i') 
two-color plane and the (V — i') versus (i' — z') plane to 
identify LBGs at (z) ~ 4 (z4LBGs) and LBGs at (z) ~ 5 
(z5LBGs), respectively. The exact selection criteria of 
the z4LBG sample are 

B-R>1.2 

20 The SDF images and catalogs are available at 
|http://soaps.naoj.org/sdf/| 



R- i'<0.7 

B-R>1.6(R-i') + 1.9, (1) 

and those of the z5LBG sample are 

V-i'>1.2 
i' - z'<0.7 

V - i' >1.8(i' - z') + 1.7 

B>3a. (2) 

These crite r ia are basically the same as those of 
lOuchi et alJ l)2004a|) but extend to fainter magnitudes. 
All magnitudes are corrected for Galactic extinction 
ijSchlegel et al.lfl998h . and all colors are measured in a 
2" aperture. In cases where the magnitude of an object 
is fainter than the la limiting magnitude, it is replaced 
with the la limiting magnitude. For the z4LBG sample, 
4543 objects down to i' tot = 17 A met our criteria, and for 
the z5LBG sample 831 objects down to z' tot = 26.6 met 
our criteria. We adopted the MAG_AUT value obtained by 
SExtractor l|Bertin fc Arnoutsl 11996') for the total mag- 
nitude. 

The completeness and contamination of th ese LBG 
samples were eval uated bvlYosh io^a ei^ all l)2006ft in a way 
similar to that of lOuchi et al.1 l|2004aj) . We carried out 
a Monte Carlo simulation creating artificial objects with 
typical high-z galaxy SEDs calculated with the model of 
iKodama & Arimotol l)1997fl . as well as the in tergalactic 
medium attenuation model of iMadaul l)1995|) . We as- 
sumed the same E(B — V) d i stribut ion as that of the 
LBG samples of lOuchi et alJ l|2004afl . These artificial 
objects were distributed on the real SDF images, then 
detected and measured in the same manner as for the 
real observed data. The completeness as a function of 
magnitude and redshift was defined as the ratio of the 
number of objects that were detected and satisfied our 
LBG color criteria to the input number of artificial ob- 
jects. On the other hand, the contamination rate was 
evaluated with the photometric redshift catalog of the 
HDF-N bv lFurusawa et all pX)0|) . We again carried out 
a Monte Carlo simulation, creating artificial objects with 
the same redshift, magnitude, and color distributions as 
the HDF-N catalog. The contamination rate was defined 
as the ratio of the number of nearby objects satisfying 
the LBG color criteria to the number of LBGs. 

So far, 22 objects from the z4LBG sample and 16 ob- 
jects from the z5LBG sample have been spectroscopically 
identified. Based on the follow-up spectroscopy, we have 
found that all of the objects that meet our criteria were 
indeed identified as high-z LBGs; i.e., the nominal con- 
tamination ra te is 0%, although the spectroscopic sample 
is still small. lYoshida et alJ l|2006D constructed another 
LBG sample at z ~ 5 using the (R — i') versus ii' — z') 
selection; however, we do not analyze this sample, as it 
would have almost the same redshift distribution as the 
z5LBG sample selected in the (V — i') versus (i' — z') 
plane, but with a larger contamination rate. 

3. uGC: THE MOCK GALAXY CATALOG 

3.1. A General Outline of the vGC 

We used a numerical mock galaxy sample to compare 
the predicted clustering strength with that of observa- 
tions. The model that we used is called the a^GC (nu- 
merical galaxy catalog: fNagashima et alJl2005j) . which is 
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implemented as a combination of two approaches, one 
a high-resolution TV-body simulation to follow the hier- 
archical merging history of dark halos, and the other 
a semianalytic method (SAM) to model some physical 
processes related to the evolution of the baryonic com- 
ponents in each dark halo. 

The TV-body simulation has a box size of lOO/i^Mpc 
on a side and treats 512 3 dark matter particles. Select- 
ing only halos containing at least 10 dark matter parti- 
cles, the z/GC follows the dark halo masses as small as 
3 x 10 9 M©, which is appropriate for this study, as it in- 
cludes the very faint and small-mass high- z galaxy pop- 
ulation. The merger trees are directly constructed by 
the A-body simulation assuming a power spectrum of 
initial density fluctuations in a Gaussian random field. 
Although individual small-mass dark halos are resolved, 
the z/GC nevertheless covers a large volume, having a 
virtual FOV of 0.66 x 0.66 deg 2 , which is about twice 
the size of the SDF. Therefore, we can compare with a 
high confidence the clustering strength of the observa- 
tions and predictions. The z/GC has the highest mass 
resolution among currently available cosmological sim- 
ulations. For example, the z/GC has a spatial resolu- 
tion 1.7 t imes higher than tha t of the "Millennium Sim- 
ulation" ijSpringel et alJl2005|> . although the box size is 
much smaller. 

Complicated galaxy formation processes can be de- 
scribed by a number of simple analytic model equations. 
In the SAM used for the z/GC, luminosities and colors of 
model galaxies were calculated with a population synthe- 
sis technique using simple stel lar populations computed 
by iKodama Ez Arimotcl l)1997(l with the Salpeter initial 
mass function from 0.1 to 60M Q , and an internal dust ex- 
tinction is evaluated from the slab models. In addition to 
galaxy mergers driven by dynamical friction, collisional 
starbursts were included during major mergers so as to 
assemble a new galaxy by converting all of the available 
cold gas into stars. The SAM of the z/GC also takes into 
account radiative cooling, supernova feedback, dynami- 
cal responses to gas removal, and tidal stripping, which 
have been shown to increase the agreement between mod- 
els and observations. 

Full details of the model itself are presented in 
iNagashima et all 1)200.^ . where it is shown that the 
model reproduces well the local luminosity function, 
color distribution, H I gas mass fraction, galaxy size 
distribution and so on. Further methodological de- 
tail is descr i bed in a couple of relevant references, 
Yaha giet alJ (2004') for the A-body simulation and 
Nagashima & YoshlJ l)2004f) for the semianalytic method. 

The mock galaxy catalog that we used for this study 
was generated from the z/GC to meet selection criteria 
almost identical to those of the SDF catalog. The mag- 
nitudes were determined with the same seeing size and 
isophotal threshold as for each band of the observations. 
The surface brightness distribution of z/GC mock galax- 
ies was confirmed to be consistent with the SDF obser- 
vations. The predictions for galaxy number counts in 
the optical bands were found to be completely consisten t 
with those observed in the SDF (Kashikawa et al. 2004). 
The predicted redshift distribution of AT-band selected 
galaxies was also found to be consi stent with the SDF 
observations <)Kashikawa et al.l|2003l K 
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Fig. 1. — Sample completeness of the z/GC mock LBG sample. 
The blue histograms show the completeness of the color-selected 
(COL-selected) uGC LBG sample, while the red lines show the 
completeness estimate of the observed SDF LBG sample. Top: 
z4LBGs in i' < 25.5 {solid line), 25.5 < i' < 26.5 (dotted line), 
and 25.5 < i' < 27 .4 (dashed line). Bottom: z5LBGs in z' < 25.75 
(solid line) and 25.75 < z' < 26.6 (dotted l i ne). T he completeness 
of the SDF LBG sample by Yoshida et al. 12006) was rcbinned in 
magnitudes from the original plot taking the average with number 
weighting. [See the electronic edition of the Journal for a color 
version of this figure.] 

3.2. The SDF LBG Mock Catalogs 

Based on the z/GC described above, high-z galaxy 
mock samples were extracted to mimic our observational 
z4LBG and z5LBG samples. However, applying the same 
color criteria to the z/GC as for the observed LBG sam- 
ples does not guarantee that the mock samples have the 
same redshift and magnitude distributions as the ob- 
served samples. We did a couple of different estimates of 
the z/GC predictions to see this uncertainty. 

First, we applied the same color selection criteria to 
the z/GC as for the actual observed SDF data to select 
LBGs at z = 4 and 5. The sample that we constructed 
in this manner is defined as the "COL-selected" sample. 
In fact, we found that our selection criteria can be em- 
ployed to isolate z — 4 and 5 galaxies correctly in the 
z/GC catalog, as is the case for the observations. This 
demonstrates that the semianalytic model imprinted on 
the z/GC predicts the luminosity and color properties 
of these high-z galaxies reasonably well. Applying the 
color criteria, 27, 456 mock galaxies were extracted to 
be compared with the z4LBG sample, and 4712 galax- 
ies with the z5LBG sample. Second, we randomly ex- 
tracted numerical galaxies from the z/GC so as to have 
completely the same selection function in redshift and 
magnitu de as that obse r ved. T he completeness estimate 
made by [Yoshida et al. (200f|) was regarded as the sam- 
ple selection function. The sample that we constructed 
in this manner is defined as the "SF-selected" sample. 
This contains 6653 and 1690 mock galaxies as the z4LBG 
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Fig. 2. — Comparison of the rest-UV color distribution for LBGs 
at z ~ 4 (top and middle) and z ~ 5 (bottom) between SDF obser- 
vations (solid lines) and the predictions of the uGC (histograms). 
The solid histograms denote the color distribution of COL-selected 
uGC LBG samples, while the dotted histograms are those of SF- 
selected i/GG LBG samples. [See the electronic edition of the Jour- 
nal for a color version of this figure.] 
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Fig. 3. — Comparison of the luminosity functions for LBGs at 
z ~ 4 (top) and 2 ~ 5 (bottom) between SDF observations by 
Yoshi da et alj 120061) (circles with error bars) and the predictions 
of the uGC (lines). The solid lines denote the luminosity functions 
of COL-selected uGC LBG samples, while the dotted lines are those 
of SF-selected ^GC LBG samples. [See the electronic edition of the 
Journal for a color version of this figure.] 



and z5LBG samples, respectively. The SF-selected sam- 
ple has exactly the same redshift and magnitude dis- 
tributions as observed, irrespective of their color crite- 
ria, while the COL-selected sample has exactly the same 
color criteria as observed, irrespective of their redshift 
and magnitude distribution. These two differently se- 
lected mock galaxy samples have the same limiting mag- 
nitudes as those of the observations in the object detec- 
tion bands, that is, i' = 27 .4 in the z4LBG sample and 
z' = 26.6 in the z5LBG sample. 

In Figure^ we show the completeness estimates of the 
COL-selected sample as a function of redshift. These 
are almost the sam e as those of the observed sample 
l)Yoshida et alJl2~0f)6fl ap art from the no r maliz ation. As 
seen in the estimate by lYoshida et alJ 1)20061) . the ac- 
tual observed catalog was affected by photometric er- 
rors, which severely decreased the sample completeness 
at fainter magnitudes. This is not the case for the nu- 
merical simulation, where there is no photometric error. 
The redshift distributions of the vGC LBG samples have 
slightly lo wer redshift extensio ns as compared to the es- 
timates of lYoshida et alJ 1)2006(1 . Although the reason for 
this discrepancy is not clear at this time, such a differ- 
ence is not unlikely because the completeness estimate 
of the observations is based on a galaxy SED model that 
is not exactly the same as that applied in the ^GC. On 
the other hand, the SF-selected sample has a selection 
function exactly identical to that of the observations, in 
which almost all (~ 93%) the members are also included 
in the COL-selected sample. 

Figure [3 shows a comparison of the rest-UV color 
distribution between the observations and ^GC predic- 
tions. The colors, corresponding to the rest-UV contin- 



uum slope, are good measures of the internal dust at- 
tenuation of LBGs. Note again that the observed colors 
have in general larger scatter than those of the simu- 
lation owing to photometric errors. The observed and 
predicted color distributions show relatively good agree- 
ment with each other, except that the model prediction 
has a slightly elongated red tail in (R — i') for z4LBGs 
and (i* — z') for z5LBGs. This agreement indicates that 
the dust ex tinction model im plemented in the vGQ works 
fairly well. Ildzi et all 1)2004)1 presented that the color se- 
lection applied to their semianalytic model reproduced 
well the observed completeness of the LBG sample at 
z ~ 4 from the GOODS survey, although the 1775 — z 850 
color did not agree completely, which is probably due 
to a deficiency of the dust amount or an incorrect ex- 
tinction adopted in the model. Our SDF LBG sample 
was found to show almost the same i' — z' color distribu- 
tion as that of GOODS, and we confirmed that the ^GC 
mock z4LBG sample has a similar i' — z' distribution with 
73% and 75% likelihoods on a Kolmogorov-Smirnov test 
for the COL and SF-selected samples, respectively. The 
^GC reprodu ces the i' — z' col or of z4LBGs much better 
than that of Ildzi et alJ 1)2004)) ; however, the color pre- 
dictions of the J'GC should still be affected by imperfect 
dust reddening, which also has a significant effect on the 
completeness estimate for the COL-selected sample. 

In Figure |31 we show a comparison of the luminos- 
ity functions of LBGs at z = 4 and 5 between the 
observation and the model predictions of the vGC 
The observed lumin osity functions were derived in 
l Yoshida et alJ (I2006D in almost the same way as in 
ISteidel et alJ (Jl999) to correct the selection function and 
incompleteness of the sample. The effective comoving 
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volume for each magnitude bin was calculated by con- 
volving the comoving volume and the completeness as 
a function of the redshift shown in Figure 2] The con- 
tamination correction was applied to the samples as a 
function of magnitude. In the predicted COL-selected 
samples, the luminosity function can be obtained in the 
same way as in the observations, although the effective 
volume was derived in the vGC on its own. In contrast, 
for the SF-selected sample, in which the observational 
selection function was used, the luminosity function was 
determined simply by dividing the numbe r counts of the 
sampl e by the effective volume derived bv lYoshida et al.1 
( 2006;) for each magnitude bin. 

The resulting luminosity functions of the COL-selected 
and SF-selected samples are denoted in Figure by solid 
and dotted lines, respectively. They are almost consis- 
tent with each other and with the observations as well; 
especially striking agreement can be seen in the z4LBG 
sample. The agreement of the COL-selected sample with 
the observations means that the simulated galaxies in 
the vGC have realistic colors at high z, while the agree- 
ment between the SF-selected sample and observation 
verifies that the z^GC predicts the same number den- 
sity and redshift distribution as is observed at high z. 
The SF-selected sample has a slightly shallower luminos- 
ity function than that of the COL-selected sample; how- 
ever, our present results for LBG clustering show almost 
no difference between the COL-selected and SF-selected 
sample, as shown below. 

4. SKY DISTRIBUTION 

Figure0|and Figure[S]show the sky distributions of our 
z4LBG and z5LBG samples, respectively. Larger circles 
denote brighter objects in total z'-band and z'-band mag- 
nitude for the z4LBG and z5LBG samples, respectively. 
The circle color represents the local surface number den- 
sity of the sample around the object. We measured the 
surface number density in a cir cle of 8/ifoMpc comov- 
ing radius around each sample object. The mean value 
and the dispersion of the number density were derived by 
the statistics of 10, 000 randomly chosen positions over 
the image. In Figure 0] and Figure the overdensities 
with 3, 2, 1, 0, —1, and < —la significances are de- 
noted with colors as indicated in the figures. The black 
shaded areas are masked regions 21 where detected ob- 
jects were rejected due to low S/N. It is strikingly 
apparent from these figures that galaxies are distributed 
inhomogeneously even in the z — 4 and 5 universe. In the 
case of the z4LBG sample, the highest number density is 
1.78(/i 70 2 Mpc 2 )^ 1 which has a 2.9a significant density ex- 



cess, while the lowest density is 0.696(/i 70 Mpc 



with 

a — 1.1a significant density drop. In the z5LBG sample, 
the highest number density is 0.458(ft- 70 2 Mpc 2 ) -1 (3.5(7 
significance) and the lowest density is 0.092(/i 70 2 Mpc 2 )~ 1 
(— 2.2cr). Note that the number density is evaluated for 
the z4LBG and z5LBG samples with different limiting 
magnitudes down to i 1 < 27 .4 and z' < 26.6, respec- 
tively. 

5. CLUSTERING ANALYSIS OF THE SDF LBG SAMPLE 
5.1. Angular Correlation Function 

21 The masked regions are defined at the SDF data release web 
site, |http:/ /soaps. naoj.org/sdf/| 
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Fig. 4. — Sky distribution of SDF z4LBG sample. Larger circles 
denote brighter objects in total i'-band magnitude. The local over- 
density, with the significance of each, is denoted by color as shown 
in the color legend. The black shaded areas are masked regions in 
which detected objects were rejected due to low S/N. North is up 
and east is to the left. [See the electronic edition of the Journal 

/°'We co te%rW^ 'ailgfe-Wpoint correlation func- 
tions (ACFs) w(9) to estimate the clustering strength 
of our photom e tric galaxy sample. We used the 
ILandv fc Szalavl l|1993f) estimator of w{9), 

DD-2DR + RR 
w ( d ) = > ( 3 ) 

where the DD, DR, and RR denote the number 
of galaxy-galaxy, galaxy-random, and random-random 
pairs having angular separations between 9 and 9 + 69. 
We generated 100, 000 random points to reduce the Pois- 
son noise in random pair counts and normalized DD, 
DR, and RR to the total number of pairs in each pair 
count. The random points were created with exactly 
the same boundary conditions as the SDF galaxy sample 
avoiding the mask regions where saturated stars domi- 
nate. The ACF can be approximated by the power-law 
form given by 

w{9) = A w 9\ (4) 

and we quantified the ACF with these two parameters, 
amplitude A w and slope (3. The measured ACFs have 
to be corrected for the "integral constraint" (IC) bias, 
which is caused by the uncertainty of the mean galaxy 
density estimated from the sample itself. We estimated 
the IC as follows: 

IC = ^ jf w(6)dn 1 dQ 2 , (5) 
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Fig. 5. — Same sky distribution as in Figure El but for the SDF 
z5LBG sample. [See the electronic edition of the Journal for a 
color version of this figure.] 



where fi is the surveyed area. Assuming that w{9) is 
denoted by Eq.(4), IC is a function of /3 and proportional 
to A w . For example, we derived IC= 0. 00279 A w for = 
—0.8. The IC was derived for several /3-values and was 
found to be a smooth function of (3. In this study, we used 
a different IC correction according to the best-fit value of 
(3, although the maximum variance of IC is only <5IC = 
0.0027. For the mock catalog, which has an FOV twice 
as wide as the SDF, we obtained a smaller IC variance 
SIC = 0.0023. 

Contamination by foreground galaxies would change 
the amplitude of the ACF. In this study, we assumed 
that these contaminating galaxies have a close to homo- 
geneous distribution. In this case, the true correlation 
amplitude A w is reduced by a factor of (1 — / c ) 2 , where 
f c is the contamination rate. The f c of ou r LBG sample 
is esti mated as a function of magnitude in lYoshida et alJ 
(200(J, and amounts to at most 20% . 

On the other hand, the completeness of the sample 
should not affect the ACF estimate, provided that the 
completeness i s dilu ted homogeneously over the FOV. 
lYoshida et alJ l|200fiD confirmed that the completeness 
distribution of our LBG sample has little variation over 
the SDF FOV and thus completeness can be assumed to 
be a function of magnitude only. We therefore did not 
correct the derived ACF in any way to take account of 
the completeness. Assuming the weak correlation limit, 
we estimated the Poissonian errors only ijLandv fc Szalavl 
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Fig. 6. — ACFs of z4LBG (top) and z5LBG (bottom) samples. 
The solid lines show the power-law fits with IC corrections. Error 
bars show the lc Poissonian errors. The dott ed lines show the ACF 
power law of the previous estimates in SDF by Ouchi ct al. ( 2004b). 
[See the electronic edition of the Journal for a color version of this 
figure ] 

1993) for the ACF as 



1 + w(0) 
DD ' 



(6) 



Actually, this estimate was confirmed to follow t he no- 
tation of Poissonian error in lArnouts et alJ |2002) quite 
well and was found to be the dominant error contribution 
to the ACF up to 100". 

The clustering strength parameter tq, of the spatial 
correlation function £ = (r/ro) 7 , is a useful parameter for 
comparisons with other studies. It can be derived fr om 
the ACF through Limber's equation ijPeeblesI 11980) if 
the redshift distribution of the sample is well known and 
F(z), the redshift dependence of the clustering strength, 
is properly assumed. In this study, we used the com- 
pleteness estimate as shown in Figure Q] as the redshift 
distribution of the sam ple and mad e the sa me assump- 
tion for F{z) as that in lOuchi etafl (|2004b|) . 

Figure H3 shows the ACF for our total z4LBG and 
z5LBG samples. The power-law fitting derived from 
Eq.(4) is represented by the solid lines, and the de- 
rived parameters are listed in Table ^ The ACF of 
the z5LBG sample shows a slightly steeper slope than 
that of the z4LBG sample. For comparison, dotted lines 
sh ow the prev i ous AC F estimates in the SDF derived 
in lOuchi et"aT] (|2004b|) with a fixed slope of p = -0.8. 
These are nearly consistent with the present results. 

5.2. Luminosity Dependence 

Next, we investigated the UV luminosity dependence 
of the LBG ACF. The sample was divided into three 
subsamples based on their z'-band magnitude and two 
based on z'-band magnitude for the z4LBG and z5LBG 
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Fig. 7. — Luminosity dependence of ACFs in the z4LBG sample. 
The ACFs of i' < 25.5, 25.5 < i' < 26.5, and 26.5 < i' < 27.4 
are represented by symbols given in the legend. The dotted line 
is the nonlinear ACF of dark matter at z = 4 calculated with 
the same observational selection function. Bottom: Our defined 
bias parameter b(9) for each luminosity subsample with the same 
symbol and color as indicated in the top panel. [See the electronic 
edition of the Journal for a color version of this figure.] 

samples, respectively. Note that these subsamples with 
different apparent magnitudes are taken to be subsam- 
ples of different rest UV luminosity because the sam- 
ple galaxies have almost the same distance from us for 
each of the z4LBG and z5LBG samples. The results 
are presented in Figure [7\ and Figure and the best- 
fit parameters of the individual subsamples are given in 
Tabled We confirmed the previous results that the clus- 
tering strength of LBGs depends on the UV luminosity 
of the sample; that is, brighter LBGs are more strongly 
clustered. The most remarkable result revealed in these 
figures is that not only the amplitude but also the slope 
of the LBG ACF has a strong dependence on luminos- 
ity in both the z4LBG and z5LBG samples, in the sense 
that brighter subsamples have higher ACF amplitudes 
and steeper ACF slopes. In other words, more lumi- 
nous LBGs have stronger clustering strengths at smaller 
scales. 

To show the significance of the result, we plot the er- 
ror contours for our two-parameter fits in Figure [5J The 
confidence levels for the fitting were computed based on 
Poissonian error statistics. The subsamples with dif- 
ferent UV luminosity have different ACF slopes at la 
and greater significance, although this is weaker than 
that for the difference in ACF amplitudes. Moreover, 
the robustness of our result, in which the clustering 
amplitude difference at small scales is essential, was 
evaluated by means of the b lockwise bootstrap method 
l)Porciani fc Giavaliscol2 002) . We first divided the whole 
SDF area into 3 x 4(= 12) subregion samples of equal 
area. We then constructed 100 artificial bootstrap sam- 
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Fig. 8. — Same as Figure but for the SDF z5LBG sample. 
The ACFs of z' < 25.75 and 25.75 < z' < 26.6 are represented 
by symbols given in the legend. [See the electronic edition of the 
Journal for a color version of this figure.] 

pies, each consisting of 12 subsamples randomly chosen 
from the whole subsample set with allowance for dupli- 
cations. The ACF was calculated for each of these artifi- 
cial bootstrap samples, and an error for each bin at small 
scales was estimated from the fluctuation among the 100 
bootstrap samples. The amplitude differences at small 
scales in luminosity-distinct subsamples were reproduced 
by this method for both the z4LBG and z5LBG samples. 
Bootstrap errors were found to be almost comparable to 
the Poissonian error, except for the smallest scale bin of 
the z5LBG sample, in which the bootstrap error was 3 
times higher than the Poissonian, due to small number 
statistics. 

It ha s been suggested by previous studies of LBG clus- 
tering llGiavalisco fc Dickinsonll200lHOuchi et"aLll2004bl 
lAdelberger et alJl2005|) that the amplitude of the ACF 
has a strong dependence on UV luminosity; however, 
these studies almost always assumed a fixed slope of 
(3 = —0.8, mainly due to the statistical uncertainty in 
small LBG samples. In this study based on a larger LBG 
sample with wider magnitude coverage, we find the first 
evidence that the ACF slope has a dependenc e on the UV 
luminosity of LBGs. iFoucaud et alJ l)2003j) concluded 
that their clustering measurements for a z ~ 3 LBG 
sample are broadly consistent with a power-law slope of 
—0.8 over all magnitudes; however, their brighter sub- 
sample (Iab = 20.0 — 23.5) has a steeper slope relative 
to that of the fainter subsample (Iab — 23.5 — 24.5), 
although the significan c e of t his difference is low. In 
iGiavalisco fc Dickinsonl 1)200 If) , on the other hand, the 
deep z ~ 3 LBG sample of the HDF was found to have 
a slightly steeper slope than that of bright ground-based 
samples, which is inconsistent with our result. However, 
this trend is quite weak within the errors, as they men- 
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Fig. 9. — Error ellipses of derived ACF parameters of A w and f3 
for luminosity subsamples of z4LBGs (top) and z5LBGs (bottom). 
The line colors are identical to FigureQand Figure|H] Ellipses from 
left to right denote from fainter to brighter luminosity subsamples. 
The solid and dotted ellipses are the lcr and 3<r confidence levels. 
[See the electronic edition of the Journal for a color version of this 
figure.] 

tioned, once both cosmic variance due to their small FOV 
and the different filter bands and different criteria for 
choosing the LBG sample for the HDF and the ground- 
based data are taken into account. 

The bias parameter, which is defined as the ratio of the 
clustering amplitudes of galaxies to that of dark matter, 
is normally measured on large scales of 8Mpc. To see 
the bias parameter variance with scale, we define the 
bias parameter b(9) as a function of scale as 

y wdm ((f) 

where w g (9) and wdm{6) are the ACFs of galaxies and 
dark matter, respectively. We cal culated wdm based 
on the nonlinear power spectrum of iSmith et al.1 ( 2003) 
with the same observational selection function as shown 
in Figure ^ I n Figure and Figure |S] Wdm is plot- 
ted in dotted lines, and b{9) is plotted in the lower pan- 
els. For the bright LBG sample, b(9) has an apparently 
increasing trend with decreasing scale, whereas for the 
faint LBG sample, b{9) has almost no dependence on 
scale. The scale dependence of the bias parameter for 
the br ight LBG sample is consistent with lHamana et ahl 
(2004), which is based on our previous SDF result in 
which bright LBGs were dominant. The scale-dependent 
bias at small scales is a generic prediction of hierarchical 
models independent of the epoch and of the model details 
(IColin et alJ lf9991. We compare this scale dependency 
with the model predictions in the next section. 

We found that the luminosity dependence of LBG 
clustering strength is more significant on smaller (< 
10") scales. The virial radius of a 10 12 M Q halo at 
z = 4-5 is roughly ~ 380hj Q kpc in a comoving scale 
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Fig. 10. — Example of multiple LBGs at z=4 on SDF. Objects 
indicated by circles in the right panels are the SDF z4LBG sample. 
B-, R-, and i'-band images are shown from left to right. Each 
image is 15" on a side. North is up and east is to the left. [See the 
electronic edition of the Journal for a color version of this figure.] 

(~ 75h? kpc in a physical scale), which corresponds 
to 10"- 12". Therefore, a large clustering difference was 
found on scales almost inside the halo virial radius. This 
scale dependence of LBG clustering recalls an implication 
that massive dark halos should contain multiple bright 
LBGs. In fact, we found pairs or multiple systems of 
bright LBGs with small (~ 10") separations in our sam- 
ple, as shown in Figure EH an d Figure ^2 Note that 
we are measuring only projected separations so far; fur- 
ther spectroscopy is required to confirm that these are 
indeed spatially close systems. It is quite often assumed 
that there is a one-to-one correspondence between LBGs 
and halos, that is, a single dark matter halo hosts a sin- 
gle gala xy, and some studies have suggested that this is 
corre ct ijAdelberger et all 119981 iGiavalisco fc Dickinsonl 
120011) . However, our results suggest the possibility of a 
large contribution from galaxy pairs in the same halo at 
high luminosities and on smaller scales. If two or more 
galaxies reside in a dark massive halo, these pairs natu- 
rally boost the ACF amplitude at smaller scales. It would 
be interesting to investigate the halo occupation distri- 
bution (HOD) formalism that statistically describes the 
relation of galaxies within a dark matter halo; however, 
such an analysis is beyond the scope of this paper. In- 
stead, we discuss these interpretations further in the next 
section in the course of direct comparisons with semian- 
alytic model predictions. 

5.3. Color Dependence 
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Fig. 11. — Same as Figure [TOl but for z=5. V-, i'-, and z'-band 
images are shown from left to right. [See the electronic edition of 
the Journal for a color version of this figure.] 

We also investigated the color dependence of the ACF 
features of our LBG samples. In this case, we use color to 
mean the steepness of the UV continuum s lope, which is 
corre lated with dust attenuation E(B — V) ijMeurer et alJ 
1999). SEP fitting to synthesized stellar popu lation 
models l|Papovich et al J 12003 iShanlev et all 12001^ sug- 
gests that more luminous LBGs are dustier. The redder 
LBGs are expected to be intrinsically brighter. To see 
the difference in clustering strength between dusty and 
less dusty LBGs of the same luminosity, we made z4LBG 
subsamples with (i' — z 1 ) > and < at i' < 26.0. At 
i' > 26.0, the shallower limiting magnitude of the z' band 
as compared to the %' band produces a color limit for the 
sample, which severely decreases the {%' — z') < sub- 
sample; thus, we did not include these faint galaxies. We 
tried to make a color subsample for t he z5LBG sample 
using the zb and zr data presented in IShimasaku et aTl 
(2005), although the zr-band data have too shallow a 
limiting magnitude to make subsamples without the color 
cut bias. 

The best-fit ACF parameters for the color subsample 
of z4LBG are presented in Tabled Although the dusty 
subsample with red colors of (i' — z') > has a slightly 
stronger clustering amplitude, it is within the la errors 
of the same value for the (i' — z') < subsample. Note 
that we here assume that redder LBGs are dustier; how- 
ever, the UV continuum slope also depends on the stel- 
lar age, and no strong evidence of large dust content 
in LBGs has been f ound from submillimeter photometry 
l)Webb et al. I I200.T). Taking accou nt of a possible corre - 
lation between age and extinction l)Shaplev et alJ l2001). 
no clustering difference between different E(B — V) sub- 



samples could refute the hypothesis that the age differ- 
ence causes the ACF slope difference. We discuss this 
later in § 7.3. 

6. COMPARISON WITH THE MODEL 
6.1. The overall ACF 

In this section, we compare our observed estimates 
of LBG clustering at z — 4 and 5 with the predic- 
tions of the hierarchical clustering model based on the 
mock SDF/LBG catalogs. As described in § 2, we con- 
structed two LBG mock catalogs, the COL-selected and 
SF-selected samples. The selection function of the COL- 
selected sample has almost no dependence on magnitude, 
as shown in Figure ^ which differs from the real obser- 
vation. Since the clustering strength depends strongly 
on the luminosity distribution of the sample, as seen 
in the previous section, it is essential to match the lu- 
minosity distribution between the model and observa- 
tions. We therefore randomly picked galaxies from the 
COL-selected sample so that the sample had the same 
normalization in selection function as observed for each 
magnitude bin. The SF-selected sample, by definition, 
had already been drawn randomly from the original i^GC 
catalog so as to have the same selection function in mag- 
nitude and in rcdshift as observed. We iterated this ran- 
dom resampling procedure 30 times, then derived corre- 
lation functions from the average value. The error for 
each scale bin, a w (8), was determined by the larger of 
either the Poissonian error estimated by Eq.(6) or the 
scatter in the random resampling. 

Figure UU shows the ACF for our total z4LBG and 
z5LBG samples, and the derived power-law fitting pa- 
rameters are listed in Table The filled circles and 
open squares denote the COL-selected and SF-selected 
samples, respectively, and the shaded regions show the 
observed ACF with ler tolerance. Both of the mock sam- 
ples show reasonable agreement with the observations at 
z = 4 and 5, although the observed ACFs have a some- 
what steeper slope than the predictions with a slightly 
stronger strength in amplitude on small scales (< 10") 
for the z4LBG sample. The two model predictions have 
almost the same clustering strength on all scales. As 
a whole, we conclude that the j^GC can reproduce the 
observed ACFs at z — 4 and 5. 

6.2. Luminosity and halo mass dependence of the ACF 

The most acute interest is in whether the model can 
reproduce the ACF slope dependence on the luminos- 
ity, as seen in our observed LBG samples. We made 
luminosity-distinct mock subsamples in the same magni- 
tude ranges as in the observed subsamples. In this case, 
we did not apply a random culling to the COL-selected 
sample as was done in the previous section in order to 
match the selection function with that of the observed 
sample. This is because the magnitude difference of the 
selection function in each luminosity subsample is negli- 
gible. We used the model galaxy magnitudes, taking into 
account internal dust extinction, for comparison with the 
observations. 

Figure^Ilshows the ACF luminosity dependence of the 
i^GC prediction for the COL-selected sample. In contrast 
to our expectations, there are almost no clustering dif- 
ferences among different luminosity subsamples. Slight 
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Fig. 12. — Model-predicted ACFs by uGC mock samples for 
z4LBGs (top) and z5LBGs (bottom). The filled circles and open 
squares denote the COL-selected and SF-selected samples, respec- 
tively. The shaded regions show the observed ACF of the SDF 
total sample with lc tolerance. [See the electronic edition of the 
Journal for a color version of this figure.] 

differences of amplitude can be seen, although they are 
much smaller than for the observations. We confirmed 
that this null result is virtually the same in t he case of 
the SF-selected sample. iWechsler et all (12001^ also con- 
cluded that semianalytic models generally show a weak 
dependence of LBG clustering on luminosity. 

A natural expectation might be that the observed clus- 
tering difference is caused by dynamical mass differences 
of the sample galaxies. To investigate this more straight- 
forwardly, we made subsamples of the LBG mock catalog 
according to the dark halo mass, Mh, determined from 
the ./V-body simulation. In this case, we accounted only 
for the limiting magnitude of the sample and did not 
correct the completeness for each halo mass subsamplc. 
In the further analysis below, we used only the COL- 
selected sample with a few contributions of low-z con- 
tamination removed to reduce uncertainties. Figure 1141 
shows the ACFs for subsamples with different halo mass 
ranges. The clustering differences are clearly noticeable. 
The ACF for higher halo masses has a steeper slope and 
stronger amplitudes at smaller scales, which is in excel- 
lent agreement with our observational results. 

Although it is in accordance with the expectation that 
more massive galaxies show stronger clustering strengths, 
it is rather surprising that the vGC model reproduces the 
ACF slope difference according to halo mass. Most hier- 
archical clustering models, which adopt a variety of star 
formation history scenarios, predict that a more mas- 
sive dark halo contai ns a larger number of galaxies with 
Nt.bq oc M°- 7 -M°- 8 (|Benson et, al.l2000tlWe"chsler et alJ 
2001). This is also confir med by smoothed p article hy- 
drodynamics simulation l)Berlind et al.ll20 03f). This re- 
lation is also expected in the vGC, which takes account 
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Fig. 13. — Luminosity dependence of ACFs in the uGC pre- 
diction of the COL-selected sample for z4LBGs (top) and z5LBGs 
(bottom). The symbols are identical to those in Figure |3 and Fig- 
ure0 [See the electronic edition of the Journal for a color version 
of this figure.] 

of collisional star formation. Figure El illustrates the 
vGC predicted occupation numbers of LBGs (Nlbg) re- 
siding in a dark matter halo. Note that in Figure 1151 
we consider only halos harboring at least one galaxy, so 
there could be halos containing no galaxies that are not 
cataloged in the i/GG. Although most of the halos have 
only one LBG, some massive halos have multiple LBGs, 
and more massive halos tend to show higher multiplicity. 
This halo mass dependence of LBG multiplicity explains 
our finding that more luminous LBGs have higher clus- 
tering amplitudes at scales as small as the virial radius 
of the dark halos. These multiple galaxies in a dark mas- 
sive halo can amplify the correlation strength effectively 
on small scales. Furthermore, the ACF of LBGs in less 
massive halos could be dominated by the contribution 
of halo-halo correlation at all the scales, which is consis- 
tent with our result of a scale-independent bias parame- 
ter for faint LBGs, as can be seen in Figure and Fig- 
ure^ Figure H~4l (top, z4LBG sample) also predicts that 
the halo-mass distinct ACFs have almost no differences 
for the low-mass halo subsample below 10 115 M Q , where 
one-to-one correspondence can be applied for all scales. 
Our observed luminosity dependence of the ACF slope 
is qualitatively consistent with the picture that massive 
dark halos harbor multiple LBGs, which contradicts the 
assumption of one-to-one correspondence in the bright 
LBG description. 

6.3. ACFs of lhalo-lLBG sample 

In the ^GC mock catalog, host dark halos can be iden- 
tified for each galaxy. To further examine the hypothe- 
sis that the steep ACF slope of the massive dark halos 
as seen in Figure [21 is caused by LBG multiplicity in 
individual halos, we made an artificial one-to-one corre- 
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Fig. 14. — Halo mass dependence of ACFs in uGC for 
z4LBGs (top) and z5LBGs (bottom). The halo mass sub- 
samples of 12.0 <log(M h /M ), 11.7 <log(M h /M Q ) < 12.0, 
11.5 <log(M h /M e ) < 11.7, 11.3 <log(M fe /M ) < 11.5, and 
log(Mjj/M ) < 11.3 are represented by symbols as shown in 
the legend, from top to bottom for z4LBGs. The mass sub- 
samples of 12.1 <log(Affc/Af ), 11.8 <\og(M h /M & ) < 12.1, and 
11.5 <log(Af^/M0) < 11.8 are represented by symbols as shown 
in the legend, from top to bottom for z5LBGs. [See the electronic 
edition of the Journal for a color version of this figure.] 

spondence mock catalog, in which all satellite galaxies 
were removed and the central brightest galaxy was left 
when a halo has multiple galaxies. In this experimen- 
tal "lhalo-lLBG" mock sample, every halo has only one 
galaxy. The ACF parameters were then derived for each 
mass-distinct subsample extracted from the lhalo-lLBG 
sample. The results are listed in Table and the com- 
parison with those derived from the original mock cata- 
log is shown in Figure [TBI In all cases, we found no ex- 
cess and sometimes a deficit of ACF amplitudes at small 
scales (< 10") for the lhalo-lLBG sample. In the case 
of deficit amplitudes, we derived the ACF parameters 
by fitting a power law at only the scales larger than 10". 
The upper panels of FigureED a PPa rcn tly reveal that the 
ACF slope, (3, does not differ among the halo-mass dis- 
tinct subsamples in the lhalo-lLBG catalog. The slopes 
of the ACFs of the lhalo-lLBG subsample coincide with 
each other around f3 ~ —0.8, which corresponds to that 
of the small halo mass subsample [log(Mh/M@) < 11.5] 
in the original mock catalog. Even a high-mass subsam- 
ple now has the same ACF slope as that of a low-mass 
subsample in the lhalo-lLBG catalog. This simple test 
clearly demonstrates that LBG multiplicity in massive 
dark halos causes the ACF amplitude excess at small 
scales, and halo occupation number plays an important 
role in the small-scale clustering of LBGs. 

Figure Hand Figure |H1 show that brighter LBGs have 
a higher amplitude even at scales (10" — 100") larger 
than the typical virial radius of dark halos (10" — 12" 
for a 10 12 Mq halo). Can this luminosity dependence of 
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Fig. 15. — Two-dimensional histograms of the dark halo mass 
and the LBG occupation number in a halo predicted by the ^GC 
for z4LBGs (top) and z5LBGs (bottom). Each color level shows 
the number of galaxies as shown in the color legend; the darker 
shaded region shows greater numbers of samples. The halo mass 
Mhalo is shown in units of 10 12 Mq. [See the electronic edition of 
the Journal for a color version of this figure.] 

clustering amplitude on larger scales be explained only 
by the halo-halo contribution? The lower panels of Fig- 
ure^Jshow the ACF amplitude A w of halo-mass distinct 
subsamples in the lhalo-lLBG catalog. They still show 
amplitude differences, although the significance of differ- 
ence is smaller than that of the original catalog, which 
means that the semianalytic model + iV-body simula- 
tion predicts that more massive dark halos are intrinsi- 
cally strongly clustered. We can conclude that the model 
predicts that the slope of the halo-halo contribution to 
the ACF is always the same irrespective of halo mass, 
whereas the amplitude at larger scales is higher for more 
massive LBGs. The observed trend in which more lumi- 
nous galaxies have steeper and higher clustering strength 
can be interpreted as a synergy of two effects: first that 
more massive halos contain multiple LBGs, and second 
that more massive halos are strongly clustered. 

7. DISCUSSION 

7.1. UV Luminosity - Halo Mass Relation 

The predicted ACF behavior of the mass-selected sam- 
ple is significantly in contrast to that of the luminosity- 
selected sample as seen in Figure EH The hierarchical 
clustering model reproduces well the observed LBG clus- 
tering difference in the halo mass-distinct sample but 
not that in the luminosity-distinct sample. To see this 
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Fig. 16. — ACF parameters, j3 and A w , of the halo-mass distinct 
subsamples in the uGC lhalo-lLBG catalog. The circles denote the 
ACF parameters of the lhalo-lLBG sample, while the tr iang les 
denote those of the original mock catalog derived in Figure [TH for 
comparison. Left: ACF slope (3 (top) and ACF amplitude A w 
(bottom) of the z4LBG sample as a function of dark halo mass. 
Right: Same as left, but for the z5LBG sample. [See the electronic 
edition of the Journal for a color version of this figure.] 

more closely, we show the relation between halo mass and 
galaxy magnitude predicted by the z^GC in Figure 1171 
The luminosity is nearly proportional to the halo mass, 
although the relation has rather large scatter. The red 
line shows the linear fit to connect the mean ridges of 
the distribution. The lower panels of Figure El show 
the halo mass distribution for each of the vGC luminos- 
ity subsamples. Although the distribution peak shifts 
slightly to lower mass as luminosity decreases, the over- 
all features of the halo mass distribution are unchanged 
with luminosity. 

Based on the mean ridge in Figure IT71 without consid- 
ering the large scatter, the hierarchical clustering model 
predicts the mass of halos hosting z4LBGs of each UV 
luminosity as follows: 

1O U - 7 M < M h < 10 12 M Q (23.5 < i' < 25.5), 

10 115 M Q < M h < 10 117 Af Q (25.5 < i' < 26.5), 

10 n - 3 M Q < M h < 10 115 M Q (26.5 < i' < 27.4). (8) 

A slightly weaker luminosity dependence of halo mass is 
seen for z5LBGs: 



10 



,11.; 



M Q < M h < lO 12 -^ (24.0 < z < 25.75), 
1 „:!."• - 



5 M< 



r < M h < 1O 118 M (25.75 < z' < 26.6). (9) 

These val ues are comparable to th e estimate for LBGs at 
z = 3 by Adcl berger et al.l 120051) and smaller than the 



estimate by Blaizotet^ al.| (|2 Q04Tl . and alm ost consistent 
with the estimate bv lOuchi et all pCK Mbl for z4LBGs at 
i 1 = 24.5, although the i^GC predicts a shallower Mh — L 
relation. 

Taking into account these correspondences of UV lu- 
minosity and halo mass in comparing the ACFs between 
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Fig. 17. — Prediction of vGC regarding the UV luminosity de- 
pendence of galactic halo mass. The magnitudes quoted are for 
the i' band in the z4LBG sample (top left) and the z' band in the 
z5LBG sample (top right). The straight lines show the linear fit to 
the mean ridge of the distribution. Bottom: Halo mass histograms 
for each luminosity subsamplc with the same lines as indicated in 
Figure 171 and Figure 1^1 [See the electronic edition of the Journal 
for a color version of this figure.] 

the observations in Figure [7| and Figure [H] and the model 
prediction in Figure the observed clustering result 
for each luminosity subsample is reconciled roughly well 
with the predictions for each halo mass subsample. A 
tighter relation between halo mass and UV luminosity 
would improve the consistency of the ACF dependence 
on luminosity between the observations and model pre- 
dictions. This conjecture is consistent with the idea that 
the observed slope difference of ACFs is due to differences 
in the occupation number of LBGs in a halo. In Fig- 
ure multiple LBGs in the same halo were inevitably 
in the same mass-distinct subsample. However, this is 
not always the case for luminosity-distinct subsamples 
in a simulated LBG sample having a UV luminosity and 
halo mass relation with a large scatter; i.e., multiple 
LBGs in the same halo could be separated into differ- 
ent luminosity-distinct subsamples. On the other hand, 
observed luminosity-distinct subsamples have the same 
clustering tendency as the mass-distinct subsamples of 
the model. Therefore, the true LBGs are supposed to 
have a tighter relation between halo mass and UV lumi- 
nosity than the model prediction. In turn, the halo mass 
plays a fundamental role in the star formation activity 
in LBGs. 

The collisional starburst model that the i/GC takes into 
account may predict a looser correlation between halo 
mass and UV luminosity that is sensitive to star forma- 
tion history and dependent on the details of the merger 
process. Moreover, the large scatter of the halo-mass — 
luminosity relation is a generic f eature of the hierarch i- 
cal clustering model suggested bylWechsler et alJ (2001). 
iSommerville. Primack fc Faberl l)2001f) also pointed out a 
weak luminosity dependence on halo mass in the semi- 
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analytic model for z ~ galaxies. This is a con- 
sequence of different star formation histor i es in diffcr- 
ent halos. iSommerville. Primack &: Faberl (|200 lh raise 
the possibility that the dust extinction increases the 
mass-to-light ratio so effectively that the model repro- 
duces the observed luminosity dependence of clustering 
strength. However, in the vQC model, the dust ex- 
tinction has been adequately included in order to repro- 
duce the observed color-magnitude relation and overall 
redshift distribution of each opt ical/NIR-selected galaxy 
sample (Naga shima et al.|l2005|) . Introducing more dust 
extinction in order to create the same strong dependence 
of mass on luminosity as observed would draw apparent 
contradictions with these observables, although the dust 
extinction in the z^GC is treated by a simple model whose 
accuracy is still uncertain. Real LBGs are expected to 
have a tighter correlation between halo mass and UV 
luminosity than expected from the semianalytic model 
prediction; however, it is not yet clear how to reproduce 
the tight correlation in the collisional starburst model, 
in which luminosity depends on the star formation his- 
tory triggered by merging in a short period. The scatter 
in the relation of halo mass and UV luminosity also sig- 
nificantly affects the shape of the luminosity function. 
Model improvements are therefore required in order to 
reproduce both the observed correlation and luminosity 
functions simultaneously. 

7.2. Comparison with Previous work 

Our new and more reliable measurements of the ACF 
for LBGs have revealed that more luminous LBGs have 
steeper ACFs especially at small scales. A h i nt of this 
signal was previously detected by Ouchi et"all 1120011) for 
LBGs at z = 4. On the contrary, 



Alle n et al.H 2005) did 



not find such an ACF steepening for LBGs at z = 4; 
however, their ACF could be measured only on scales 
larger than 10", so their results may be due to prefer- 
entially detecting the brightest central galaxy in a halo 
with their wide and shallow (i < 24.5) data. The steep- 
ening of the AC F for more lum i nous galaxies at z = 3 
was predicted bv lKravtsov et all (|2004t see their Fig. 12), 
which suggests that this is due to a more pronounced 
one-halo component dominated by galaxy pairs within a 
single halo. Our results of observation and the compar- 
isons with a semianalytic model for luminous and mas- 
sive LBGs at z = 4 and 5 are qualitatively in agreement 
with the implications of their work. However, if such a 
steepening on small scales is a universal trend in high-z 
galaxies, the clustering analysis made on a small contigu- 
ous FOV would detect only their one-halo components, 
leading to an incorrect measurement for the overall clus- 
tering strength at th e epoch. The actual prediction of 
iKravtsov et alJ l)2004|) is not the ACF steepening but the 
departure from a single power law of the spatial corre- 
lation function. Based on deep imaging data, we have 
so far obtained only the ACF, which is the projection of 
the spatial correlation function, and therefore the steep- 
ening that we observe could be a result of dilution of 
their predicted trend. Our present observational data 
have insufficient statistical resolution to allow the de- 
tection of the depart ures from a single power law that 
IKravtsov et alJ l|2004[) predicted for the correlation func- 
tions of high-z galaxies. Such a departure can be seen 
in a more statistically robust LBG sample l|Ouchi et alJ 



12005(1 . A future large spectroscopic survey for high-z 
LBGs will derive the precise spatial correlation function, 
in which we could easily verify the noticeable transition 
scale from the two-halo term to the one-halo term. 

7.3. Dependance of Clustering on Age of Population 

Although we have shown that the observed ACF slope 
difference can be accounted for by LBG multiplicity in 
a single halo, here we consciously discuss an alternative 
possible interpretation to explain the observational re- 
sult. An amplitude enhancement of the bright LBG ACF 
has certainly been observed on small scales within the 
typical virial radius of a dark halo; however, this could 
be a projection effect owing to seeing an excess of halo- 
halo pairs on more extended spatial scales. Such an ef- 
fect can be expected, especially when there is large-scale 
structure traced by bright LBGs almost along the line 
of sight. Although the CDM theory predicts that seeds 
of more massive dark halos collapse earlier with inherent 
stronger clustering, there could be an additional effect 
irrespective of dark halo mass to enhance the clustering 
difference between bright and faint LBGs. If the lumi- 
nous LBGs are an older population than faint LBGs, 
the observed ACF difference could b e explained as a 
bias d ependency on formation epoch. Yoshikaw a et all 
l)2001[) found a clear tendency that early-forming galaxy 
populations are distributed tightly with the underly- 
ing mass density field, while the late-forming popula- 
tion tends to be less concentrated. This is basically 
caused by stochastic biasing, so the formation of late- 
forming galaxies is suppressed in overdense and high- 
temperature regions where it is hard to cool gas effec- 
tively. The tendency for a rich environment to favor the 
early formation of galaxies can be seen in the p rotoclus- 
ter regions of z - 2 LBGs ijSteidel et al.N2005l). Similar 



numer ical results have been obtained by Blanton et alJ 
(1999) and lSommerville. Primack fc Faberl (|2001^l . which 



explain the clustering dependence on galaxy color seen in 
the local universe represented by the morphology-density 
relation. The predicted correlation functions show obvi- 
ous slope differences between old and young populations. 
The same explanation could be applied to our finding of 
ACF slope differences in LBGs. The essential assump- 
tion in this interpretation is that brighter LBGs should 
be older, which seems to be plausible from the observa- 
tional implication that UV-luminous LBGs a t z = 3 have 
larger stellar masses l|Papovich et al.ll2002j) . However, 
Shanlev et al.l l)2001j) found a contradictory tendency in 
that the UV luminosity is uncorrelated with the inferred 
stellar mass for the brighte r LBG sample. On the other 
hand, Shaplev et alJ l|2001f) found a possible correlation 
between dust extinction and age for z ~ 3 LBGs, al- 
though this depends strongly on the applied attenuation 
law. If this is the case, our finding of no clustering dif- 
ference between different E(B — V) subsamples as seen 
in § 5.3 could be an argument against the hypothesis 
described above that the age difference causes the ACF 
slope difference. More studies are required to support 
this hypothesis. 

8. CONCLUSIONS 

We explored the clustering properties of LBGs at z = 4 
and 5 with an angular two-point correlation function 
on the basis of the very deep and wide Subaru Deep 
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Field data. We constructed mock LBG catalogs based 
on a semi-analytic approach to the hierarchical cluster- 
ing model combined with a high-resolution iV-body sim- 
ulation, carefully mimicking the observational selection 
effects. The luminosity function for LBGs at z = 4 and 
5 predicted by this mock catalog was found to be consis- 
tent with the observations. Our main conclusions about 
the LBG clustering properties are as follows. 

1. We confirmed the previous results that the clus- 
tering strength of LBGs depends on the UV luminosity 
range of the sample; that is, brighter LBGs are more 
strongly clustered. 

2. We found apparent slope differences among UV 
luminosity subsamples of LBGs at both z = 4 and 5. 
More luminous LBGs have steeper correlation functions. 
The bias parameter, defined as the ratio of the clustering 
strength of galaxies to that of dark matter, was found to 
be a scale-dependent function for bright LBGs, while it 
was almost scale-independent for faint LBGs. Luminous 
LBGs have a high bias at smaller angular scales, which 
decreases as the scale increases. 

3. The overall clustering strengths of LBGs at z = 4 
and z = 5 are reproduced reasonably well by the hierar- 
chical clustering model when the observational selection 
function is taken into account. The observed clustering 
difference according to UV luminosity is not apparent in 
the model. However, the halo-mass distinct samples were 
able to reproduce this difference. More massive LBGs 
have steeper correlation functions. The model predicts 
that LBG multiplicity in a massive dark halo makes this 
ACF slope difference, and halo occupation number plays 
an important role in the small-scale clustering of LBGs. 

4. Our finding that there is a luminosity dependence 
of correlation function slope is probably an indication 
that massive dark halos must host multiple bright LBGs. 
This is also supported by our semi-analytic model, which 



predicts that more massive dark halos have a higher oc- 
cupation number of LBGs. 

5. The hierarchical clustering model could reconcile 
the observed luminosity dependence of the ACF when 
there is a tight relation between UV luminosity and dark 
halo mass. 

Our finding of a large difference in clustering strength 
on small scales will provide new insights into the sub- 
structure inside dark halos, on which we have unsolved 
questions: how many galaxies reside in a single halo, 
and how are they distributed and how do they evolve 
in a halo? Interestingly, these questions are also raised 
by more precise measurements of clustering in the lo- 
cal universe, in which significant departures from a sin- 
gle power-law ACF on a small scale ha ve been revealed 
l|Zenavi et all 12004 iPhleps et all 12005ft . The nonlinear 
behavior of the bias parameter depending on scale is not 
surprising because the bias on small scales should depend 
on several complicated processes of interhalo dynamics, 
including cooling, star formation, supernova feedback, 
photoionization, merging, and tidal disruptions. More 
precise measurements of galaxy clustering on small scales 
are required from upcoming observations of both the lo- 
cal and distant universe, and in addition, refinements of 
the semi-analytic approach with high resolution would 
help in better understanding the detailed galaxy forma- 
tion and evolution processes in a dark halo. 
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Lyman Break Galaxy Clustering 



TABLE 1 

ACF Parameters of SDF LBG Samples 



Sample 




/c(%) b 


/3 C 




r d 


z4LBG total 


4543 


2.6 


-0 qn+ ' 06 
u - yu -0.04 


9 O1+0.44 
Z,oi -0.4B 


4 69+ - 81 

4 - Dy -o.67 


z4LBG i' < 25.5 
z4LBG 25.5 < i' < 26.5 
z4LBG 26.5 < i' < 27 .43 


916 
1977 
1650 


0.38 
2.3 
5.1 


i rjc+u.iy 

1 ' ZO -0.25 

-o.o8±S:il 

-0.80l° ;i« 


10 44+ 6 - y9 
3.59ll-g 

1 48+ 1 ' 07 
l-48_ 7g 


6 52 +4 Ua 

5.38_^ 

4 09+ 2 -' 77 
4uy -1.66 


z4LBG («' - z') > 0.0 
z4LBG (V - z') < 0.0 


1124 
653 




-i-08±»:ig 

-1 04+ - 50 


5 - y4 -1.96 
2.47+^ 8 5 


c -, ,, + 2.12 
6 - 14 -1.58 
4 lq+13.7 
4 - iy -3.05 


z5LBG total 


831 


10 


-1 02 +lU4 


7.16+^ 


6.09+f^ 


z5LBG z' < 25.75 
z5LBG 25.75 < z' < 26.62 


325 
506 


9.0 
11 


_1 15+U.lb 
-0 85+ ' 27 


18.68+ 5 ;« 
3.031^7 


8.161-.- 
4.78±I; 2 3 * 



a Number of sources in the sample. b Contaminat ion rate rebinned in magnitudes 
from the original estimate bv lYoshida et al. (2006) taking the average with number 
weighting. c The ACF power-law fitting parameters defined in Eq.(4) with contami- 
nation correction. 11 The correlation length inferred from ACF. 



TABLE 2 

ACF Parameters of uGC LBG Samples 



Sample 






p 




z4LBG total COL-selected 
z4LBG total SF-selected 


12712 
6653 


5.6 
0.0 


n 74 +0.04 
u - '*-0.06 

-o.74i8:8! 


0.83+8- 2 ! 
1 26+° :33 


z4LBG i' < 25.5 
z4LBG 25.5 < i' < 26.5 
z4LBG 26.5 < i' < 27 .4 


4979 
10735 
11742 


4.6 
6.1 
6.0 


-o.8ol8:85 

-0 7fi+°- 02 
u -' D -0.04 
n o 9 +0.04 


T OO+0.28 
L -°°-0.33 

76+ - 11 
U -' D -0.10 

Q5+ - 10 
u - yo -0.16 


z4LBG 12.0 <log(JW h /M Q ) 
z4LBG 11.7 <log(M h /M ) < 12.0 
z4LBG 11.5 <log(Afh/M ) < 11.7 
z4LBG 11.3 <log(M h /M Q ) < 11.5 
z4LBG log(A/ h /M ) < 11.3 


2928 
4556 
5174 
7201 
5983 


0.0 
0.0 
0.0 
0.0 

0.0 


i - dJ -0.01 
i 09+O.O6 
-■-••"-0.26 

-0 Q2+ - 08 
u - y ^-0.04 

-0 77+ - 11 
u -' '-0.03 

-o.64i8:« 


44 1Q+ U - 49 

^• ly -0.49 

13.32±;-« 

2 m+°- 39 

69+ - 14 
u.oa_ 26 

^7+°- 21 


z4LBG lhalo-lLBG 12.0 <log(Af h /Af ) 
z4LBG lhalo-lLBG 11.7 <log(M h /M Q ) < 12.0 
z4LBG lhalo-lLBG 11.5 <log(Af h /A/ ) < 11.7 
z4LBG lhalo-lLBG 11.3 <log(M h /M Q ) < 11.5 
z4LBG lhalo-lLBG log(M h /M G ) < 11.3 


1796 
3783 
4931 
7135 
5960 


0.0 
0.0 
0.0 
0.0 
0.0 


-o.82i8:i 2 
-o.88i8:i 2 
-o.8oi8:ig 
-0.8618:18 
-0.8518:18 


9 oq + 2. 13 
/ - 88 -1.22 
2 00+ 1 ' 47 

o.96i8;l| 
0.8718^2 

67+ - 59 
u - D '-0.37 


z5LBG total COL-selcctcd 
z5LBG total SF-selected 


2603 
1690 


0.5 
0.0 


-o.8il8:8 7 
-o.86i8:^ 


2-06l8;S 

2.56ll: 8 8 


z5LBG z' < 25.75 
z5LBG 25.75 < z' < 26.6 


1611 
3101 


0.12 
1.2 


-0.84_ 10 
_n 74+O.O6 
u - '^-0.06 


o -17+1.40 

01 ' -1.20 
-, 09+O.42 
1,0 -0.36 


z5LBG 12.1 <log(Af h /Af ) 
z5LBG 11.8 <log(M h /A/ ) < 12.1 
z5LBG 11.5 <log(Af ft /M ) < 11.8 


440 
1117 
2547 


0.0 
0.0 
0.0 


_1 24+ u - u « 

-1 04+ 06 

-CI 90+ 08 
u - yu -0.08 


49.701^ 
13.40l 2 :^ 

2 40+ - 91 
z - 4u -0.74 


z5LBG lhalo-lLBG 12.1 <log(Af h /Af ) 
z5LBG lhalo-lLBG 11.8 <log(M h /Af ) < 12.1 
z5LBG lhalo-lLBG 11.5 <log(Af h /A/ ) < 11.8 


385 
1029 
2486 


0.0 
0.0 
0.0 


-0.84_ Q 28 
-0 64+ - 18 
-0 72+ - 18 


7.59+ 5 4 77 

2.09ii:?i 

-, 00+0.86 

1 -°°-0.72 



a N and f c are average values for the random resampling process for the total and luminosity 
subsamples. 



